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During  the  period  April.  15,  1988  -  May  15,  1989,  AFOSR  supported 
research  work  on  the  the  stochastic  behavior  of  the  LMS  and  related 
adaptive  algorithms  has  yielded  results  in  two  major  areas: 


The  mathematical  models  developed  in  [15]  were  extended  to  analyze  the 
behavior  of  four  types  of  digital  implementations  of  the  basic  LMS 
algorithm. 

a)  Nonlinear  quantization  effects  in  LMS  and  Block  LMS  adaptation  were 
compared  on  the  basis  of  dynamic  range,  algorithm  transient  response  and 
stalling  phenomena.  It  was  shown  [3]  that  the  LMS  algorithm  requires 
(1/2  Log  2  L  -  K  )  fewer  bits  than  the  BLMS  algorithm  for  the  same 
saturation  and  stalling  effects  (L=block  length  and  K  lies  between  .2  and  1). 

b)  Saturation  effects  in  the  LMS  Adaptive  Line  Enhancer  (ALE)  were 
studied  using  the  saturation  model  developed  in  [15].  The  ability  of  a 
digitally  implemented  ALE  to  cancel  a  weak  sinusoid  in  the  presence  of  a 
strong  sinusoid  and  noise  was  investigated  [4].  The  mathematical  model 
and  the  simulations  both  showed  a  significant  slowdown  in  cancellation  of 
the  weaker  sinusoid  when  the  larger  sinusoid  caused  the  algorithm  to 
operate  in  saturation. 

c)  [15]  studied  the  effect  of  a  saturation  non-linearity  on  the  error  term  in 
the  weight  up-date  term  in  LMS  adaptation.  [15]  was  extended  to  consider 
the  effect  of  a  saturation  non-linearity  on  the  entire  weight  up-date  term 
[5].  By  comparison  with  the  [15],  there  is  no  significant  difference  in  the 
behavior  of  digital  implementations  of  the  LMS  algorithm  whether  round¬ 
off  occurs  before  or  after  multiplying  the  error  by  the  data. 

d)  [15]  was  also  extended  to  the  study  of  LMS  Echo-Cancellation  when  the 
data  is  non-gaussian  [6].  It  had  been  previously  shown  that,  when  the 
LMS  algorithm  is  implemented  using  a  sign  detector  (one  bit  quantizer) 
for  the  error,  serious  performance  degradation  occurs  for  binary  data  as 
compared  to  gaussian  data.  The  saturation  nonlinearity  of  [15]  was  used  to 
study  the  performance  of  the  binary  and  gaussian  data  models.  The 
number  of  additional  bits  for  binary  data  for  the  same  performance  was 
evaluated. 


in-Ngi§g 


i  RLS  Performance  Comparison  for  Tracking  a  Chirped  Sinusoid 


The  RLS  adaptation  algorithm  is  well-known  to  have  a  faster 
transient  response  than  the  LMS  algorithm  when  the  input  data  is  colored 
and  stationary.  It  is  not  clear  that  RLS  is  superior  to  LMS  for  tracking 
non-stationary  inputs.  The  chirped  sinusoid  is  an  example  of  a  non¬ 
stationary  input  for  which  it  is  useful  to  obtain  a  performance  comparison 
of  the  two  algorithms. 

[8]  studies  the  ability  of  the  LMS  adaptive  algorithm  to  track  a  fixed 
amplitude  complex  chirped  exponential  buried  in  additive  white  gaussian 
noise.  The  exponential  is  recovered  using  an  M-tap  predictor  W  (adaptive 
line  enhancer).  When  W  is  controlled  by  the  LMS  algorithm  with  forget¬ 
ting  rate  v  =  pPn  (Pn  is  the  input  noise  power),  the  output  misadjustment 
is  dominated  by  a  lag  term  of  order  v-2  and  a  fluctuation  term  of  order  v. 
A  value  Vopi  exists  which  yields  a  minimum  misadjustment  Vopt  and 

i^min  were  evaluated  as  a  function  of  the  signal  chirp  rate  the  number 
of  taps  M,  the  noise  power  Pn  and  the  signal-to  noise  ratio  p. 

[7]  studies  the  ability  of  the  exponentially  weighted  RLS  adaptive 
algorithm  to  track  a  fixed  amplitude  complex  chirped  exponential  buried  in 
additive  white  gaussian  noise.  The  exponential  is  adaptively  recovered 
using  an  M-tap  predictor  W.  Five  principal  results  of  this  work  are:  1)  the 
performance  of  the  algorithm,  2)  the  methodology  of  the  analysis,  3)  proof 
of  the  quasi-deterministic  nature  of  the  data-covariance  estimates,  4)  new 
analysis  of  RLS  for  an  inverse  system  modelling  problem,  and  5)  new 
analysis  of  RLS  for  a  deterministic  time-varying  model  for  the  optimum 
filter.  It  is  shown  that,  when  W  is  controlled  by  the  RLS  algorithm  with 
forgetting  rate  p,  the  output  misadjustment  is  dominated  by  a  delay  term 
of  order  p-2  and  a  fluctuation  term  of  order  p.  Thus,  a  value  Popt  exists 
which  yields  a  minimum  misadjustment  min-  Popt  ^nd  ^re 

evaluated  as  a  function  of  the  signal  chirp  rate  v,  the  number  of  taps  M, 
the  noise  power  Pn  and  the  signal-to  noise  ratio  p. 

The  results  in  [7]  and  [8]  were  combined  in  [9]  to  yield  tracking 
performance  comparisons  between  the  two  algorithms.  The  minimum 
misadjustments  for  the  two  algorithms  were  compared  for  the  same  set  of 
input  signal  parameters.  For  a  satisfactory  predictor  output  signal-to-noise 
ratio,  LMS  will  track  better  than  RLS  (smaller  misadjustment)  unless  p>>l 
or  v  is  sufficiently.  Thus,  this  work  leads  to  the  very  useful  and 
unexpected  result  that  LMS  usually  tracks  better. 


POTENTIAL  AIR  FORCE  APPLICATIONS 


A.  The  LMS  algorithm  is  an  extremely  popular  form  of  adaptation  because 
of  its  simplicity  of  implementation  and  well-understood  behavior.  One 
would  assume  that  it  has  found  use  in  many  Air  Force  systems  where  a 
priori  statistical  information  about  the  input  signals  is  unavailable.  These 
applications  could  include  echo-cancellation  for  hard-wire  communication 
systems,  adaptive  interference  cancellation  for  spread-spectrum 
communication  systems,  adaptive  beam-forming,  and  adaptive  noise 
cancellation  in  jamming  environments.  In  each  case,  digital  implementa¬ 
tions  of  the  algorithm  require  an  understanding  of  the  number  of  bits 
required  for  representing  the  individual  mathematical  operations  that 
comprise  the  algorithm.  The  results  of  the  work  described  in  Section  A 
help  a  designer  to  select  the  correct  system  parameters  for  fast  and 
efficient  implementations  of  the  algorithm. 

B.  When  can  one  use  the  simpler,  yet  more  slowly  converging,  LMS 
algorithm  in  place  of  the  faster  converging,  but  significantly  more  difficult 
to  implement,  RLS  algorithm?  This  analysis  suggests  that  LMS  is  superior 
to  RLS  when  tracking  some  non -stationary  signals.  The  impact  of  this 
result  on  adaptive  systems  used  by  the  Air  Force  would  be  to  retain  the 
simplicity  of  LMS  for  tracking  environments. 
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A  Weighted  Normalized  Frequency  Domain  LMS 

Adaptive  Algorithm 

SHAUl.  FI.ORIAN  and  NEIL  J.  BERSHAD.  ff.iiow,  ieee 


AAornrr— Thh  pnprr  prr<criilv  •  icrnrral  filif riiiK  srhrmr  for  nhlaln- 
inE  an  inpiil  pnwrr  rcliinalr  fur  «rlllns  Ihr  rnnvarErnrr  paramrlrr  p 
vrpnralrl.v  in  rarh  rrrqiirnr)  bin  nf  a  rrrqiirnr;'  rfiiinain  I.MS  adapliva 
nilrr  d-'DAI'')  ale«rilhni.  A  linear  fillrrinp  npcrallon  la  prrfnrmad  nn 
Ihr  mafniliidr  Aipiarr  nf  Ihr  Inpiil  data  and  inrnrporatrd  dirrril)'  intn 
Ihr  alEnrilhin  a<  a  dala-drprndrni  timr-var.vine  Mnrhasllr  p(n). 

1'hr  inran  prrfnrinanrr  »f  Ihr  nriphlrd  nnrnialir.rd  frrqurnry  do¬ 
main  I.MS  alforithin  (WNFDAI'')  is  anniyrrd  iisina  I.I.d.  (laiissian  data, 
and  Ihr  rrsiills  art  validated  hy  Ihr  Monir  Carlo  simulations  of  Ihr 
alKorithm. 

1'hr  sinnilalions  arr  also  lisrd  lo  study  Ihr  wrlKhl  transirni  brhav- 
hir.  1’hr  sinnilalions  siiEsrsI  that  short  smiHilhinE  limes  arr  suflirirni 
for  rapid  wriRhl  ronrrrEriirr  wilhnul  larpr  flurtualions  in  Ihr  ponrr 
rsliniairs  siEniliranlly  alTrrtlnE  IransIrnI  wrlKhl  lirhavinr. 


I.  Introdiichon 

REQIJENCY  dnmnin  implemcntalinn  of  the  LMS 
ailaplive  filler  has  advantages  over  lime  implementa- 
cions.  Improved  convergence  properties  and  reduced 
computational  complexity  are  the  two  main  advantages 
H1.121. 

In  applications  of  the  frequency  domain  LMS  adaptive 
filter  where  the  input  oower  varies  dramatically  over  dif¬ 
ferent  frequency  bins,  some  input  power  measurement 
must  be  incorporated  in  the  algorithm.  A  recent  paper  j.^l 
described  a  normalized  frequency  domain  adaptive  LMS 
filler  (NFDAF).  The  normalization  involved  an  estimate 
of  the  input  power  in  each  frequency  bin  using  a  uni¬ 
formly  weighted  moving  average  window.  The  power  es¬ 
timate  was  included  directly  in  the  frequency  domain  LMS 
algorilhm.  The  statistical  behavior  of  the  NFDAF  in  (3] 
was  investigated  using  Gaussian  data  models.  Closed  form 
expressions  were  derived  for  the  transient  and  steady-state 
mean  weight  and  mean-square  error  per  bin. 

The  piirpo.se  of  this  paper  is  lo  extend  the  analysis  to  a 
weighted  NFDAF  (WNFDAF)— an  adaptive  filter  which 
incorporates  an  arbitrary  linear  weighting  on  the  magni- 

M:iiiiis('tipl  rcixivcil  M:iuli  26.  I9R6;  revised  J.inuary  I*),  1988  This 
«otl  wiis  supported  hy  the  II  S  Air  ToriT  Otliic  of  Scirntiric  Resesreh 
iimlrr  Comt.-irl  Al'DSR  86  t8t9,1.  Proicrl  2VVI/A6. 
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tude  square  of  the  data  for  input  power  estimation.  Thus, 
this  extended  theory  is  useful  for  the  hoimalized  fre¬ 
quency  domain  LMS  filter  with  power  estimates  that  have 
recursive  descriptions  and/or  nohunifonri  weighted  mov¬ 
ing  average  |3)-|6].  A  further  purpose  of  this  paper  is  to 
study,  via  Monte  Carlo  simulations,  the  transient  behav¬ 
ior  of  the  WNFDAF  when  the  initial  settings  of  the  power 
estimate  are  varied. 

These  results  are  a  measure  of  how  rapidly  the  algo¬ 
rilhm  can  respond  to  changing  input  power  levels— a 
problem  which  was  not  investigated  in  (3]. 

II.  Analysis 
A.  Mathematical  Model 

Using  the  FDAF  model  in  [3],  if  the  input  processes  to 
the  FFT's  are  widc-sense  stationary  over  the  observation 
lime,  then  disjoint  spectral  outputs  are  uncorrelated.'  As¬ 
suming  the  inputs  are  joint  Gaussian  random  processes, 
the  disjoint  bins  of  the  FFT  output  prbvide  statistically 
independent  outputs.  Thus,  each  complex  tap  is  operating 
on  independent  data.  Furthermore,  since  the  FFT  opera¬ 
tions  are  linear  operations  on  the  joint  Gaussian  input  se¬ 
quences,  the  FFT  outputs  are  jointly  complex  Gaussian 
sequences. 

The  weight  update  equation,  corresponding  to  i  single 
complex  lap.  is  given  by  (7) 

W{n  -f  I )  =  H'(n)  -P  pe{n)  x*{n) 

=  W{n)[]  -  pljr(n)l*)  +  gu/(n) jr*(n) 

(') 

where 

W(n)  =  complex  scalar  weight  on  the  nth  iteration 
e(n)  =  error  waveform  =  d{n)  —  y{tt) 
y{n)  =  filter  output  =  H'(n)jt(rt) 
rf(n)  =  reference  waveform 
jr  ( n )  =  input  data  sequence 
p  =  adaptation  coefficient. 

The  adaptive  filter  scheme  incorporates  the  same  input 
signal  model  as  in  |3).  The  input  sequences  <f(n)  and  xin) 
contain  a  desired  component,  buried  in  statistically  inde- 

'Alilirnigh  ihi*  is  inic  for  white  noise  inputs,  ii  Is  not  generally  tnie  since 
ihe  FPT  bins  are  not  spectrally  disjoint.  However,  the  bins  are  apptoxi- 
mntrly  unrnrrrialed  it  the  pr  r  .spectrum  of  Ihe  signal  cKsngel  slowly 
over  the  bandwidth  of  a  single  FFT  bin  |l| 
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On  the  Probability  Density  Function  of  the  LMS 

Adaptive  Filter  Weights 

NEIL  J  BERSHAD,  fellow,  ieee.  and  LIAN  ZUO  QU 


AbilracI — In  this  paper,  the  Joint  probability  density  function  of  the 
weight  vector  In  LMS  adaptation  is  studied  foi  Gaussian  data  models. 
An  exact  expression  Is  derived  for  the  characteristic  function  of  the 
weight  vector  at  time  n  -l-  1  conditioned  on  the  weight  vector  at  lime 
n.  The  conditional  characteristic  function  Is  expanded  In  a  Taylor  se¬ 
ries  and  averaged  over  the  unknown  weight  density  to  yield  a  0rst- 
order  partial  dilferential-diirerence  equation  in  the  unconditioned 
characteristic  function  of  the  weight  vector. 

The  equation  is  approximately  solved  for  small  values  of  the  adap¬ 
tation  parameter  and  the  weights  are  shown  to  be  Jointly  Gaussian  with 
time  varying  mean  vector  and  covariance  matrix  given  as  the  solution 
to  well-known  difference  equations  for  the  weight  vector  mean  and  co¬ 
variance  matrix.  The  theoretical  results  are  applied  to  analyxing  the 
use  of  the  weights  In  detection  and  lime  delay  estimation.  Simulations, 
which  support  (he  theoretical  results,  are  also  presented. 

1.  Introduction 

HE  time  domain  LMS  adaptive  filter  algorithm  [I]  has 
found  many  applications  in  situations  where  the  sta¬ 
tistics  of  the  input  processes  are  unknown  or  changing. 
These  include  noise  cancelling  [2],  line  enhancing  (3}, 
HI,  17],  and  adaptive  array  processing  (5],  16).  The  al¬ 
gorithm  uses  a  transversal  filter  structure  driven  by  a  pri¬ 
mary  input  (Fig.  1).  The  filter  weights  are  updated  itera¬ 
tively  based  upon  the  difference  between  the  filter  output 
and  a  reference  input,  so  as  to  minimize  the  mean-square 
error  of  the  difference. 

The  LMS  algorithm  has  been  very  thoroughly  investi¬ 
gated  over  a  long  period  of  time.  Its  transient  weight  mean 
and  covariance  matrix  and  mean  square  ‘•rror  behavior 
have  been  evaluated  precisely  l6]-[9]  for  uncorrelated  in¬ 
put  data.  Expressions  for  the  transient  mean  square  error 
as  a  function  of  the  eigenvalues  of  the  data  covariance 
matrix  Rxx  have  been  given  by  a  number  of  authors  for 
both  the  real  [6],  [8],  (9]  and  complex  [6],  (7]  LMS  al¬ 
gorithms. 

Although  the  results  in  16]-I9]  are  useful  for  determin¬ 
ing  the  transient  and  steady-state  mean  and  covariance  of 
the  LMS  algorithm,  there  are  many  situations  where  ad¬ 
ditional  statistical  information  about  the  weight  vector 
would  be  useful.  These  cases  include  detection  of  a  nar- 
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row-band  line  componeht  in  background  noise  using  the 
weight  vector  as  a  test  statistic  (Adaptive  Line  Enhancer 
17),  I10)-[14)),  by  using  the  filtered  output  as  a  test  sta¬ 
tistic  [13],  [14],  and  for  time  delay  estimation  [IS]- 
More  recently,  the  LMS  algorithm  has  been  used  as  a 
canceller  as  part  of  a  spread  spectrum  communication  sys¬ 
tem.  The  output  of  the  canceller  acts  as  the  input  to  a 
matched  filter  binary  decision  device.  Knowledge  of  the 
statistics  of  the  canceller  output  is  crucial  to  predicting 
error  probabilities  for  the  system  [16],  [17],  [28], 

In  all  four  cases,  knowledge  of  the  first  and  second  mo¬ 
ments  of  the  weights  is  not  sufficient  to  calculate  1)  re¬ 
ceiver  operating  characteristics  (ROC’s)  relating  detec¬ 
tion  and  false  alarm  probabilities  for  the  detectors,  2)  error 
probabilities  for  the  binary  decisions,  and  3)  estimation 
performance.  Most  often,  via  a  central  limit  argument,  it 
is  assumed  that  the  test  statistic  and/or  the  weights  are 
Gaussian  [14],  [16],  [17],  In  some  unpublished  work  [18], 
simulations  suggest  that  the  weights  are  Gaussian.  How¬ 
ever,  there  is  no  existing  theory  supporting  these  simu¬ 
lations.  Some  recent  theoretical  work  [19]  on  the  fre¬ 
quency  domain  LMS  algorithm  has  shown  that  the  single 
complex  weight  is  Gaussian  in  steady  state.  Further  recent 
work  has  shown  this  result  also  holds  in  the  transient  case 
after  an  initial  phase  of  adaptation  [20], 

In  this  paper,  the  joint  characteristic  function  of  the 
weight  vector  in  real  LMS  adaptation  is  investigated  when 
the  inputs  are  zero  mean  stationary  Gaussian  sequences, 
independent  from  iteration  to  iteration.  Simulations  which 
support  the  theoretical  analysis  are  also  presented.  The 
results  of  the  analysis  are  applied  to  I)  detecting  correla¬ 
tion  between  the  primary  and  reference  inputs  using  the 
weight  vector  as  the  input  to  a  matched  filter,  and  2)  es¬ 
timating  the  relative  delay  between  the  primary  and  the 
reference. 
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Abstract 


Analog  implementations  of  the  LMS  and  Block  LMS  (BLMS)  adaptive  filtering 
algorithms  have  been  shown  to  be  equivalent  with  respect  to  adaptation  speed 
and  steady-state  misadjustment  errors.  However,  the  BLMS  algorithm  offers 
significant  reductions  in  computational  complexity  due  to  block  processing. 

In  this  paper,  digital  implementations  of  the  two  algorithms  are  compared 
with  respect  to  finite  word  effects.  The  algorithm  stalling  phenomena  is 
studied  using  gausslan  data  models  and  conditional  expectation  arguments. 

It  is  shown  that  the  BLMS  algorithm  requires  (•“•  log^  L  -  k)  fewer  bits  for 
the  same  stalling  behavior  (L  =  block  length  and  K  lies  between  .2  and  1, 
depending  on  the  precise  definition  of  algorithm  stalling).  On  the  other 
hand,  the  LMS  algorithm  requires  log2L  fewer  bits  than  BLMS  for  the  same 
level  of  saturation  behavior  (transient  response)  at  algorithm  initialization. 
Hence,  overall  the  LMS  algorithm  requires  ^/^og2L+K)  fewer  bits  than  the  BLMS 
algorithm  for  the  same  saturation  and  stalling  effects. 
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ABSTRACT 


Digital  Implementation  of  the  LMS  Adaptive  Line  Enhancer  (ALE)  Introduces 
certain  nonlinear  effects.  Ihls  paper  investigates  feedback  error  signal 
saturation  effects  on  the  ALE  adaptation.  A  set  of  non-llnear  coupled 
difference  equations  is  derived  by  projecting  the  mean  weight  vector  upon  a 
set  of  orthogonal  basis  functions.  •  These  equations  are  used  to  study  the 
transient  behavior  of  ALE  for  the  case  of  one  and  two  sinusoids  in  broadband 
noise.  Simulations  are  presented  which  support  the  results  of  the  theoretical 
model. 
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ABSTRACT 


The  effect  of  a  saturation  type  non-llnearlty  In  the  weight  update 
equation  In  LMS  adaptation  Is  Investigated  for  a  white  gausslan  data  model. 
Non-llnear  difference  equations  are  derived  for  the  weight  first  and  second 
moments  that  Include  the  effect  of  a  1-e  *  saturation  type  non-llnearlty  on 
the  update  term  driving  the  algorithm.  A  non-llnear  difference  equation  for 
the  mean  norm  Is  explicitly  solved  via  a  differential  equation  approximation 
and  Integration  by  quadratures.  The  steady-state  second  moment  weight 
behavior  Is  evaluated  approximately  for  the  nonlinearity.  Using  these 
results,  the  tradeoff  between  the  extent  of  weight  up-date  saturation,  steady- 
state  excess  mean-square-error  and  rate  of  algorithm  convergence  Is  studied. 
For  the  same  steady-state  mlsadjustment  error,  the  trade-off  shows  that 
1)  starting  with  a  sign  detector,  the  convergence  rate  Is  Increased  by  nearly 
a  factor  of  two  for  each  additional  bit,  2)  as  the  number  of  bits  Is 
Increased  further,  the  additional  bits  buy  very  little  In  convergence  speed, 
asymptotically  approaching  the  behavior  of  the  linear  model.  Thus,  by 
conparlson  with  previous  results  [3],  there  is  no  significant  difference  In 
the  behavior  of  digital  Implementations  of  the  LMS  algorithm  whether  round-off 
occurs  before  or  after  multiplying  the  error  by  the  data. 
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Abstract 

The  effect  of  a  saturation  type  error  non-linearity  in  the  weight 
update  equation  in  LMS  adaptive  echo-cancellation  is  investigated  for 
an  independent  binary  data  model.  A  nonlinear  difference  equation  is 
derived  for  the  mean  norm  of  the  difference  between  the  estimate  and 
the  unknown  filter  to  be  estimated  by  the  algorithm.  The  difference 
equation  is  evaluated  numerically.  It  is  shown  that  far-end  binary 
data  interference  is  much  more  deleterious  to  algorithm  transient  be¬ 
havior  than  far-end  gaussian  data  interference.  The  number  of 
additional  bits  for  the  same  cancellation  convergence  rates  for  binary 
vs.  gaussian  interference  of  the  same  power  is  studied  as  a  function  of 
various  system  parameters. 

Algorithm  convergence  rates  are  studied  as  a  function  of  an 
arbitrary  probability  density  function  for  the  far-end  data.  It  is 
shown  that  a  binary  pdf  causes  the  worst  degradation  and  a 
gaussian  shaped  pdf  causes  the  least  degradation. 
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ABSTRACT 


This  paper  studies  the  ability  of  the  exponentially  weighted  RLS 
adaptive  algorithm  to  track  a  complex  chirped  exponential  signal  buried  in 
additive  white  gaussian  noise.  The  signal  is  adaptively  recovered  using  an 
M'tap  predictor  W. 

There  are  five  principal  results  of  this  paper:  1)  the  performance  of 
the  algorithm,  2)  the  methodology  of  the  analysis,  3)  proof  of  the  quasi- 
deterministic  nature  of  the  data-covariance  estimates,  4)  new  analysis  of 
RLS  for  an  inverse  system  modelling  problem,  and  5)  new  analysis  of  RLS 
for  a  deterministic  time-varying  model  for  the  optimum  filter.  Specifically, 
it  is  shown  that, 

1 )  when  W  is  controlled  by  the  RLS  algorithm  with  forgetting 
rate  ^  =  (1-X),  the  output  misadjustment  is  dominated  by  a 
delay  term  of  order  and  a  fluctuation  term  of  order  p. 
Thus,  a  value  Pqpj  exists  which  yields  a  minimum 
misadjustment  :W  “'***.  P^pj  and  :W*“*”are  evaluated  as  a 
function  of  the  signal  chirp  rate  v,  the  number  of  taps  M,  the 
noise  power  and  the  signal-to  noise  ratio  p.  For 
sufficiently  small  y, 

P„p,=  {(M  +  5^"’*"=  (3/4)  P„(M  +  l)p„p,. 

2)  The  estimate  of  data  covariance  matrix  R(k)  satisfies 


lim  E{[R(k)-E[R(k)]]^}<<{E[R(k)]}^ 
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ABSTRACT 


This  paper  studies  the  ability  of  the  LMS  adaptive  algorithm  to  track 
a  fixed  amplitude  complex  chirped  exponential  buried  in  additive  white 
gaussian  noise.  The  exponential  is  recovered  using  an  M-tap  predictor  W. 
When  W  is  controlled  by  the  LMS  algorithm  with  forgetting  rate  v  =  pPjj 
(Pq  is  the  input  noise  power),  the  output  misadjustment  is  dominated  by  a 
lag  term  of  order  v'^  and  a  fluctuation  term  of  order  v.  Thus,  a  value 
exists  which  yields  a  minimum  misadjustment  and  are 

evaluated  as  a  function  of  the  signal  chirp  rate  \|/,  the  number  of  taps  M, 
the  noise  power  and  the  signal-to  noise  ratio  p.  For  sufficiently  small  \|/, 


These  results  are  new  and  important  because  they  represent  precise 
analysis  of  a  non-stationary  deterministic  inverse  modelling  system 
problem.  These  results  are  in  agreement  with  the  form  of  the  upper 
bounds  for  the  misadjustment  provided  in  [4]  for  the  determistic  non- 
stationarity. 


iicv-.c.  \fiK'i  A-.^-’}-^''-  r.efcvv^ 

SUPERIORITY  OF  LMS  OVER  RLS  FOR  TRACKING  A  CHIRPED  SIGNAL 
Odile  Macchi^  and  Neil  Bershad* 


I.  Introduction 

When  an  adaptive  filter,  receiving  coloured  inputs,  has  to  track  a  nonstadonary  environment, 
it  is  often  said  that  the  RLS  algorithm  will  outperform  the  LMS  one  because  it  is  known  to  converge 
faster.  However,  convergence  speed  is  a  transient  property,  independent  of  the  amount  of  noise, 
while  tracking  is  a  steady-state  performance  and  is  therefore  also  influenced  by  the  noise  level.  The 
answer  is  not  obvious,  lliere  is  in  fact  no  single  answer,  and  it  depends  very  much  on  the  problem 
under  consideration.  It  has  already  been  prov^  [1]  that  LMS  can  ^  superior  in  a  context  where  the 
optimal  filter  Wo(k)  to  be  tracked  is  a  zero-mean  random  funcdon  of  time. 

In  this  contriburion  we  consider  a  case  where  W^fk)  has  determinisdc  time  variations.  A 
coherent  signal  s(k)  with  power  Pj  is  buried  in  additive  white  noise  n(k),  with  power  Pn.  It  is  known 
that  the  SNR  can  be  improved  by  implementing  a  predictor  based  on  past  observed  samples 


X(k)  =  (x(k-l), ...,  xfk-M)^;  x(k)  =  s(k)  +  n(k)  (1.1) 

ft(k)  =  §(k)  =  WT  X(k).  (1.2) 

In  fact,  if  we  consider  the  errors 

c(k)4x(k)-^(k),  (1.3) 

il(k)4s(k)-  §(k),  (1.4) 

c(k)=il(k)  +  n(k),  (1.5) 


the  optimal  estimator  Wq  minimizing  E(ln^l)  is  at  the  same  time  the  optimal  predictor  minimizing 
E(leP),  the  noise  sequence  being  independent 

When  the  sinusoid  s(k)  is  subject  to  chirping  with  a  chirp  rate  yf,  according  to 

s(k)  =  VZPgexp  i(k9  +  y  v  +  ?),  (1.6) 

the  optimal  filter  Wo(k)  is  time-varying.  It  therefore  has  to  be  tracked,  for  example,  by  means  of  an 
adaptive  algorithm  of  the  kind 

I 

W(k)  =  W(k-l)  +  F(e(k),  X(k) ...),  (1.7) 

e(k)  4  x(k)  -  W(k-1)T  X(k).  (1.8) 

The  system  is  depicted  in  Rg.l.  This  specific  example  occurs  in  practical  HF  radars. 

The  filter  input  x(k-l)  is  coloured  since  it  contains  a  sinusoid.  We  are  thus  in  the  exact 
situation  where  it  has  always  been  assumed  that  RLS  will  outperform  LMS.  In  what  follows  we 
show  that  this  assumption  is  true  in  the  case  of  a  high  input  SNR,  but  in  the  situation  of  a  poor  SNR, 
it  is  wrong:  LMS  will  outperform  RLS  unless  the  chirp  rate  w  is  very  high.  The  limiting  case 
corresponds  to  a  fixed  value  c  less  than  1  of  the  so-called  "non-stationarity  degree",  defined  by 
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ABSTEACT 

The  Recuralve  Lcaat  Squarea  (RLS)  algorlthn 
ia  known  to  converge  faater  than  the  Leaat  Hean 
Squarea  (LHS)  algorlttai  when  the  environaent  ia 
atationary  and  the  input  ia  colored.  It  ia  then 
often  concluded  that  RLS  will  track  better  than 
LHS  in  a  non-atatlonary  environaent.  Aa  an 
exaaple  of  a  non-atationary  colored  input,  thia 
paper  atudlea  a  chirped  alnuaold  a  burled  in 
additive  white  nolae  n.  a  can  be  adaptively 
recovered  ualng  an  H-tap  predictor  W.  When  W  ia 
controlled  by  the  RLS  algorltha  with  forgetting 
rate  B  ~  (1-X),  the  output  Blaadju8taent2 
la  dominated  by  a  delay  tera  of  order  0 
and  a  fluctuation  tern  of  order  B.  Hence,  a 
value  B  exlata  which  ylelda  a  alninun  value. 
Sinllar'^Bihavlor  by  the  LHS  algorltha  reaulta  in 
another  nlniaun  nlaad juatnent.  The  ratio  of  the 
alniaa  doea  not  depend  on  the  chirp  rate  f) 
but  on  H  and  the  input  SNR  p.  For  a  aatiafactory 
predictor  output  SNR,  LMS  will  track  better  than 
RLS  (aaaller  aiaadjuatnent)  unleaa  1)  P  >>  I  or 
2)  i  la  aufflclently  large.  Thua,  thia  work 
leada  to  the  very  uaeful  and  unexpected  reaulc 
that  LHS  uaually  tracka  better! 

I.  IMTEODUCTIOM 

Adaptive  algorlthna,  which  are  capable  of 
tracking  a  non~atatlonary  input,  alwaya  exhibit 
two  contradictory  fcaturea;  (i)  the  convergence 
apeed  -  a  tranaient  property  which  laptovea  aa 
the  forgetting  rate  (aay  B)  Increaaea,  (11)  the 
ateady  atate  fluctuatlona  -  due  to  neaaurcnent 
and  algorltha  noiae  which  degrade  the  perforaance 
aa  B  Increaaea.  It  la  often  aaaimed  that  RLS 
tracka  a  nonatatlonary  environaent  better  than 
LMS  becauae  it  haa  a  greater  convergence  apeed. 
However,  thia  aaeuaption  la  not  aufflclently 
Juatlfied  becauae  ii)  above  la  not  conaidered. 
Indeed,  tracking  ia  not  a  tranaient  problea  but  a 
ateady-atate  problea.  Thua,  the  aeaaureaent  and 
algorltha  noiae  are  of  primary  iaportance.  tn 
thia  paper,  a  correct  methodology  la  preaented  to 
deal  with  the  tracking  coapariaon  of  RLS  and  LMS 
for  the  non-trlvlal  caae  of  a  nolay  chirped 
ainuaold.  The  alnuaold  la  to  be  recovered  by  an 
adaptive  predictor.  For  both  algorlthaa,  the 
aiaadjuatnent  or  teaidual  aean  aquare  error  of 
the  adaptive  filter  output  la  properly  alnlalxed 
by  optiaizlng  the  forgetting  rate.  The  reault- 
Ing  B  ,11  dependa  on  the  noiae  and  the 
chlrp)B|  raBi^(  v  ia  the  atep-aize  of  LHS)  . 

The  correapondlng  alaadjuataenta  and 

are  then  be  compared  aa  a  function  of  ti  Pt  and 
N.  In  the  aoet  Intereating  caae  (email  p, 
average  f)  LMS  la  found  auperlor. 

Thia  work  waa  aupported  by  the  U.S.  Air  Force 
Office  of  Scientific  Reeearch  under  Project 
2304/A6,  Grant  86-(X)93,  by  the  National  Center 
for  Scientific  Reeearch  (CNRS),  Covernaent  of 
France  and  the  Unlveralty  of  Parle,  Oraay. 
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II.  aitncwRRiiic  A  MOIST  CBUFSS  SKSIAL  " 

- 

For  a  coherent  algnal  a(k)  burled  lA  ahlte 
noiae  n(k),  the  SNR  can  ba  iMprovad  by  adding  A 
nunber  of  aaaplea  H  properly  phaae  COrteCted  by, 
a  tranaveraal  filter  W.  Thia  ia  the  baalB  Idea 
behind  the  "adaptive  line  enhancer"  ahovn  te  .•\5 
Fig.  1.  The  device  la  adaptive  because  tlM 
predictor  weights  are  controlled  by  the  (AOtay). 
predictive  error  e(k)»  The  folloaing  AOtatiOa  Ed 
used:  i  i  .  1 W"!  ' 

noisy  algnal:  x(k)“a(k)  ♦  n(k)  (l»l) 

past  aaaplea:  X(k)“(x(k-1),... ,x(k-M)J^  .. (J»l) 

predicted  signal:  a(k)  •  If^X(k)^ 
recovery  error:  n(k)  "  a(k)  —  e(k)  (1*4) 

control  error:  e(k)  “  x(k)-M^X(k)  ()*^) 


recovery  error: 
control  error: 


n(k)  -  a(k)  -  e(k) 
e(k)  -  x(k)-M‘^X(k) 


n(k)  4  n(k)  .  (lift) 


The  noiae  sequence  n(k),  with  yower  F^,  «a.*y.aw^ 
•••tned  Independent^  Vti#  10  eeeuMd  C6 

be  e  chirped  elnusoldg  ntmmXf  *^4  . 

.(k)  -  /r  axp  JJkO  ♦  4  ♦  ♦ 

where  9  la  a  randoa  phase  with  flat 
density,  0  is  related  to  the  cantat  fregaMadiy 
and  ♦,  hereafter  called  "chirping  fata’f  Id  •  ‘  ^ 
reaponalble  for  the  non-atatlonary  Charaelat  Or*'  . 
the  signal;  P^  is  the  signal  power  and  tha  la^C  ‘ 
SNR  is 

Clearly  Eq.  (2.7)  cannot  held  for  all  M  , 

(-•,  4-1.  However,  It  ia  a  taasonabla  Mdal.^v;. 
Inside  a  time  Interval  T  aueh  that  |t9|  «  ••  il 
Due  to  Eq.2.6,  alnlalzatlon  bf  tha  raeodaTy  anW 
g(|n|^}  la  equivalent  te  ainlaltatlen  »f  tbd  y-r 
control  error  B(|e|  )•  Tha  alnlaltatlOB  6aa 
performed  recuralvely,  ualng  a(k)  td  CMtfal  IM 
filter  state  V,  using  either  tha  RLS  (add  IdetM 
111)  or  the  LMS  algorltlai  (aaa  Section  IT)* 

Consider  the  optlaal  filter  W  (k) 
varying  when  9  la  nonzero)  which  onMld  bd  •  1  ,  , 


achieved  ideally  by  both  algorlthaa. 
Using  the  notation 

. .  .,«(•>*.  . .1"*) 


|bCg  be 
b  -  exp  J(jj; 


....  bV*l* 

c  -  a»p  l-j  |) 


(r.lo^ 

Clilt) 


It  haa  been  ahown  In  (3]  that  tha  optlaal  (llCM* 

la  given  by  ^  .a  .aiv 

N  (k)  -  RV  D  (f*ll)' 

o 


R  S  p/(l4Mp) 


(I.IS) 
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La  plupart  des  6(udes  sur  I'adaptation  LMS  en  annulation  (ficho  considferent  des  donnfes  gaussiennes  el/ou  on  algorithme  ope¬ 
rant  en  mode  lindaire.  La  rdalitd  est  quelque  peu  diffdrente:  -  Timplanlation  numdrique  des  algorithmes  introduit  des  non  lindaritds, 
-  les  donndcs  sont  souvent  binatres.  Dans  cet  article,  un  parambtre  de  saturation  contrfile  une  non  lindaritd  agissant  sur  reneor  <fii 
Intervient  dans  I'algorithme  LMS.  Cette  non  lindaritd  peul  alnsl  varier  de  fa;on  continue  de  la  fonction  signe  (quandneateor  ft  1  W) 
ft  la  fonction  lindaire  (absence  de  quantification).  Les  rdsultats  mesurds  par  I'dcho  rdsiduel  montrenl  que,  compardes  ft  des  donndes 
gaussiennes,  des  donndes  binaires  ddgradent  rTautant  plus  les  performances  que  la  saturation  est  dievde  (proche  de  la  fonedon  rigne). 
La  diffdrcnce  entre  les  nombres  de  bits  de  I'erreur  nfcessaires  pour  des  donpdes  gaussiennes  et  binaires  est  dvalude  pour  on  mime 
dcho  rdsidi'el  en  fonction  de  la  puissance  des  donndes  lointaines  et  du  parambtre  de  saturadon. 


The  effect  of  a  saturation  type  error  non-linearity  In  the  weight  update  equation  in  LMS  adaptive  echo-cancellatioo  is 
invesdgated  for  an  independent  binary  data  model.  A  nonlinear  difference  equation  is  derived  for  the  mean  nom  of  the  diffefeoce 
between  the  esUmate  and  the  unknown  Filter  to  be  estimated  by  the  algorithm.  The  difference  equation  is  evaluated  numetkally.  It 
is  shown  that  far-end  binary  data  Interference  is  much  more  deleterious  to  algorithm  transient  behavior  than  far-end  gaussian  data 
interference.  The  bit  differential,  for  the  same  performance  in  a  digital  implementation  of  the  algorithm,  is  studied  for  binary  vs 
gaus^n  data  as  a  function  of  the  binary  data  power  and  the  saturation  parameter  of  the  non-linearity. 


i 


I  INTRODUCTION 

Quelques  articles  rbcents  [1-5]  ont  btudib  Talgorithme  LMS 
modifib  par  la  presence  de  non  linbaritbs  dans  I'incrbment. 
Dans  [2,3]  des  non  linbaritbs  de  type  saturation  sont  utilisbes 
pour  modbliser  les  effets  dus  ft  la  prbeision  Hnie  dans  des  im- 
plbmentations  numbriques  de  I'algorithme  LMS.  L  "hypothlse 
de  donnbes  gaussiennes  faite  lors  de  ces  btudes  n'est  plus  trbs 
rbaliste  dans  Ic  cas  d’annulation  d'bcho  de  donnbes. 

Le  cas  de  donnbes  binaires  en  annulation  d'bcho  a  bib  btudib 
dans  [4,5]  lorsque  la  fr>nction  signe  apparalt  dans  I'algorithme 
LMS.  II  est  montrb  que  par  rapport  ft  ^es  donnbes  gaussiennes, 
des  donnbes  binaires  de  mime  puissaince  dbgradent  les  perfor¬ 
mances  de  I'annulation  d'bcho  (pour  un  mime  pas  d'adaptation). 
Tel  n'est  pas  le  cas  pour  I'algorithme  LMS  classique  oft  la 
densitb  de  probabilitb  des  donnbes  n’intervient  pas. 

Cel  article  btend  les  rbsultats  de  [2,4].  II  analyse  les  effets 
des  donnbes  binaires,  sur  la  vitesse  de  convergence  de 
I'algorithme,  lorsque  le  nombre  de  bits  utilisbs  pour  reprbsenter 
I'erreur  qui  apparalt  dans  I'incrbment  de  i'algorithme  varie  de  1 
(fonction  signe)  ft  I'intini  (cas  linbaire).  Les  rbsultats  permet- 
tent  notamment  d'bvaluer  le  nombre  de  bits  supplbmentaires 
nbcessaires  lorsqu'on  passe  du  cas  gaussien  au  cas  binaire  pour 
une  mime  performance  i'annulation  d'bcho. 

II  ANALYSE 

1.  Algorithme  LMS  modifib 

Le  probibme  do  I'annulation  d'bcho,  illustrb  par  la  figure  1, 
reprbsente  un  systbme  do  transmiss[pn.b|Jatbrale  de  donnm  [4], 


Le  rbcepteur  doit  reconstituer  les  donnbes  lointaines  do(n)  ft 
partir  d'une  observation  d(n)  qui  est  la  somme  de  d«(n),  tTun 
bruit  additif  ng(n)  et  d'un  signal  d'bcho  des  donnbes  prodies 
x(n).  Cet  bcho  s'fcrit  WjX(n)  oft  est  un  Filtre  htcomni  de 
longueur  N  et  X(n)  le  vecteur  des  donnbes  proches 

X^(n)  « (x(n),  x(n  - 1),  x(n  -  2) . x(n  -  N  +  I)). 

Pour  supprimer  I'bcho  on  utilise  un  filtre  adaptatif  W(n).  Le 
signal  d'erreur  ft  I'entibe  du  rbcepteur  peut  s'bcriie 

c(n)  =  do(n)  +  n,(n)-VT(n)X(n)  0) 

oft 

V(n)«'W(n)-'Wo.  (2) 

On  suppose  que  les  termes  de  I'bquation  (1)  soot 
indbpendants  deux  ft  deux.  Les  donnbes  proches  x(n)  soM 
indbpendantes,  binaires,  de  puissance  A  et  la  longueur  N  du 
nitre  adaptatif  est  suffisamment  grande  pour  que  la  sortie 
V''^(n)X(n)  du  filtre  diffbrence  V(n)  soit  considbfbe  coRWue 
gaussienne  [4], 

Le  signal  d'erreur  qui  sert  ft  piloter  les  coefficients  de 
I'annuleur  est  aussi  le  seui  signal  dont  dispose  le  rbcepteur. 
Une  restitution  correcte  de  ne  pourra  dotK  fttie  obtenne  qaipifts 
une  bonne  rbduction  de  I'bcho  rbsiduel  V^(n)  X(h>.  Om  peut 
btre  obtenu  par  Valgorithme  LMS  de  pas  dftdiqitatlon  |i: 

W(n  4  I)  ■  W(n)  4  p  e(n)  X(n),  (3) 


